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ABSTRACT 

The quality of a literacy program can be assessed by 
taking into account the following factors: purpose of the program, 
how the program meets the most important current needs of the adult 
population, if it is accessible to the people who need services, and 
whether the literacy content is relevant to learners' needs. 
Flexibility of funding also is important. Assessment can be carried 
out on two levels — program evaluation and learner assessment. 

Criteria for funding a literacy program include the following: Does 
this program have the particular emphasis determined for current 
literacy funding priorities? Is this a high-quality program, 
according to "Model Indicators" by the U.S. Department of Education? 
Is the program achieving the particular purpose and specific goals it 
was set up to achieve? Is this an established, proven, high-quality 
program? or Is this a new program that has promise? The following are 
some best practices in literacy: using the most current teaching 
methods, including the contextual approach, the learner-centered 
approach, the participatory approach, and the use of new learning 
technology. (This paper includes a synopsis of the U.S. Department of 
Education Model Indicators of Program Quality.) (KC) 
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What Makes A Quality Literacy Program? 

One Worth Funding? How Do We Evaluate That Quality? 

TO ASSESS THE QUALITY OF A LITERACY PROGRAM 

T Tssess program quality, you will want to consider the following factors: 

The purposeof the literacy program. Quality is measured, in part, by how well the program is meeting 
its stated purpose. For example, is the purpose: 

• To serve those who have not been reached by other literacy programs in your area? 

• To improve literacy skills among children in public schools by working with care-givers and 
children? 

• To help learners achieve immediate functional goals (for example, obtain a better job)? 

• To help develop motivation, habits and skills for lifelong learning? 

Quality can be evaluated at two levels.You can evaluate quality at the organization or overall 
program level (management, infrastructure) and at the service level (strengthening literacy). In 
literacy, these two levels are called (1) PRCXjRAM EVALUATION and (2) LEARNER AS.SESS- 
MENT. Learner assessment is the evaluation of how learners are meeting their learning needs, while 
program evaluation is the evaluation of the overall program which includes learner assessment. 
Elements of program quality evaluation include community needs assessment, program planning 
processes, board involvement and effectiveness, learner involvement, attrition rate, long-term impacts 
on learners, and cost-effectiveness. Elements of learning quality include: interviews with learners 
about their goals and needs, learner feedback on teaching approaches and materials, learners’ 
assessment of their progress in meeting their goals, a display of what learners produced and tests of 
educational gains. 

In quality programs, both program evaluation and learner assessment are ongoing processes; both 
include all players, valuing and using learners’ feedback as much as staffs and administrators’; and 
both observe process as well as effectiveness based on purpose and standards. They are used not only 
to keep stakeholders informed about a program’s progress but also to improve the program. Both can 
be derived through tw'o methods: quantitative and qualitative evaluation. Quantitative indicators tend 
to be more objective and numerically measurable, and can be standardized across sites (such as drop- 
out rate or pre-/post-test scores). Qualitative indicators tend to be more subjective, tailored to measure 
outcomes in the unique learner or program, and measurable through comparisons (such as review of 
program plans, or interviews with learners about how well the program meets their needs). To evaluate 
quality, it is useful to look for both quantitative and qualitative measures of program evaluation and 
learner assessment. 
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FROM UNITED WAY DIRECTORS: WHAT MAKES A LITERACY 
PROGRAM ONE WORTH FUNDING? 

“Asa first step, I make sure the program fits what our United Way has defined as priority funding. For 
example: Does t . program reach a large audience through a public-television program accompanied 
by print materials, or a small audience through one-to-one personalized individualized tutoring? Does 
it focus on adults reading below the 6th-grade level, or on youth in our public high schools who need 
remedial tutoring? Does it teach literacy skills for immediate use, or for the long term (leam-how-to- 
leam skills)? As a second step, 1 make sure the program has a good track record; I prefer to look at grade- 
level advancement based on standardized tests. As a last step, if I still can’t decide between worthy 
programs. I’ll look at other factors: how cost-effective they are, how they judge their own performance, 
whether they have trained providers who can motivate learners.” (Charles Colvin, Montgomery Area 
United Way, Alabama — Metro II) 



“Our criteria for funding literacy programs: (1) That the program meets the most important current 
needs of our adult population— and these needs change. (For example, we are losing our major 
employer, the Air Force ba.se, and there will be new literacy needs for these displaced employees.) 
(2) That it is accessible to the people who need services (time of day, and geographically— especially 
important for us in our large rural county). (3) That the literacy content is relevant to learners’ ‘real life 
literacy needs.” (Susanne Sandusky, United Way of Aroostook, Maine — Metro VII) 

“I look for the responsiveness of the proposal to those needs identified in our community — who needs 
literacy most and where they are; v/here the gaps in services are. I also look for sensitivity to ethnic and 
cultural practices, and for accessibility to learners who are hajdest to reach. I do use quantitative 
evaluation, such as average length of stay in the program, but I think qualitative evaluation is just as 
useful; for example: asking learners questions about their programs.” (Sharon Morioka, United Way 
of San Diego, California — Metro I) 

“The flexibility of funding is important: We must allow opportunities for creativity and fund newer 
innovative programs that demonstrate viability — as well as the proven and established programs. 
Programs that lack steady funding resources also should be given funding consideration.” (Jeannette 
Morrison-Marks, United Way of Oakland County, Michigan — Metro II) 

CRITERIA FOR FUNDING A LITERACY PROGRAM 

In assessing whether or not you want to fund a literacy program, the following criteria may be helpful: 

Does this program have the particular emphasis determined for our literacy funding priorities? 

(For example, “This year, our United Way will fund literacy for public-housing residents — or 
job-related literacy — or family literacy.”) 



Is this a quality program, according to Model Indicators by the U.S. Department of Education? 
(See summary below.) 



Is the program achieving the particular purpose and specific goals it was set up to achieve? (How 

do you know? By using qualitative and quantitative e\'almtion.) 

Is this an established, proven program — clearly one of quality — which our funding will serve 

to strengthen? (How do you know? By using qualitative and quantitative evaluation.) 

Is this a new program or a promising approach with evidence of quality which our funding will 

serve to launch, develop and/or demonstrate? (See below for examples of promising 
approaches.) 

BEST PRACTICES IN LITERACY— EFFECTIVE APPROACHES WORTH 
FUNDING 

Each community has a variety of programs to meet different t>ves of literacy needs. Although there 
are many different types of literacy programs — Adult Basic Education (ABE), English as a Second 
Language (ESL), native language literacy, General Equivalency Diploma (GED) preparation, family 
literacy and workplace literacy, to name a few — what they all have in common are their instructional 
approaches. The following approaches represent the most effective — those at the “cutting edge,” in the 
view of many literacy professionals. Most programs combine two or more of these approaches. 

• Contextual (or “Functional Context”) Approach. Literacy isdevelopedwitnintheccntext 
of real-life applications and needs of the learner (for example, using the actual tasks and 
materials used on the job). Of the four approaches included here, this approach was derived 
from research on how adults learn best. The functional context approach is more motivating 
to learners; results in more rapid and lasting learning; achieves double gains (needed content 
and needed communications skills); and can decrease instructional time and thus program 
costs. The federal government is increasingly promoting this approach. 

• Learner-Centered Approach. In this approach, learning themes and materials are selected 
according to learners’ stated needs and preferences. The appioach builds on learners’ 
strengths — what they already know — rather than on “fixing” deficiencies as defined by 
others. It uses everyday materials that learners bring in or choose themselves and which are 
culturally and ethnically relevant to learners. Learner-developed materials are utilized: 
newsletters, plays and stories). Learners are the experts concerning their own programs and 
are actively involved in such program and instructional decisions as identifying program 
standards and outcomes and individualized goals. 

» Participatory Approach. This approach is characterized by learners being actively involved 

in selecting topics and materials. Lecturing occurs only when necessary and is followed by 
a critical discussion of the lecture. There is a “rule of thumb” that 90 percent of the speaking 
in class should be by learners, and only 10 percent by teachers. Learners teach and learn from 
each other, and there is integration of experiential, hands-on activities such as internships in 



schools or field trips to job sites. Interactive projects are completed by small groups within 
a larger class. Formaximum interaction, there is a small teacher/leamerratio, nevermore than 
1 to 15. 



• Use ofNew Learning Technologies. F ' ' ational technology is not a substitute for teaching; 
it does not always reduce learning time, .. requires staff training and it can cost more money 
than it saves. However, new technologies can be motivating, fun and effective as learning 
tools, especially when used in conjunction with the approaches listed above. Use of new 
technologies enables individualization and self-pacing, and builds skills transferrable to work 
settings. Examples of new technologies include computer software, interactive videodisc, 
audio tind videocassette recorders, pocket calculators and television programs, usually 
accompanied by print materials. Computer-assisted instruction (CAI) is the most widely used. 



RECOMMENDED RESOURCES 

• Model Indicators of Program Quality for Adult Education Programs. Office of Vocational 
and Adult Education, U.S. Depanment of Education, July 1992. Call Tammy Fortune, (202) 
205-9996, to order multiple copies of this free 23-page pamphlet which lists the eight 
indicators (reprinted below on this fact sheet) and information about how they were developed. 



• Assessment of Literacy Needs and Resour ces,\5n\Xtd'^diy oi 1990. Thisexcellent 

tool provides survey questions that a volunteer team can use to identify the types and quality 
of literacy programs in your community, and organizations’ infrastructure to support the 
program. This booklet was sent to all United Ways. To order more, just call. 



• For information on program evaluation or on helping modify your state’s quality indicators, 
contact the Association for Community Based Education (ACRE), (202)462-6333. For two 
excellent articles on program evaluation and learner assessment, contact the Business Council 
for Effective Literacy (BCEL), (212) 512-2415, for their January 1990 newsletter, and the 
ERJC Clearinghouse, (800) 848-4815, for Adult Literacy Education: Program Evaluation 
and Learner Assessment (Document No. 338). For information on new technologies, call the 
Adult Literacy and Technology Project, c/o Lucy MacDonald, The Fearless Future, (503) 399- 
5242. 



Many local United Ways have found that local partnerships and collaborations are an effective strategy 
for strengthening literacy in a community. To learn more about building literacy program partnerships 
andcollaborations, see the United Way of America booklets SelectedModels of Literacy Collaboration, 
Communities Working Collaboratively for Change, and Community Initiatives in Literacy. To learn 
more about workplace literacy, see the UWA booklets, A Literate Workforce: What Business Can Do 
and United Way Involvement in Workplace Literacy. To learn more about family literacy, see the 
accompanying UWA fact sheet. 



U.S. DEPARTMENT OF EDUCATION MODEL INDICATORS OF 
PROGRAM QUALITY 

The National Literacy Act, passed by Congress in 1991, called for development of indicators of 
program quality that could be used as models to judge program effectiveness. Federal funding for 
literacy appropriated through this act will be distributed by the states, which will evaluate literacy 
programs based on these model indicators (unless other indicators are developed by the states by July 
1 993). Knowing about these indicators may benefit you and literacy programs in your area in two ways: 
You may increase the quality of your programs, and you may be competitive for National Literacy Act 
funding. You can use these eight indicators (developed by consulting literacy nractirioners and experts) 
as a guide for funding programs or for providing technical a.ssistance for program improvem.ent. 

Educational Gains 

Indicator!: Learners demonstrate progress toward attainment of basic skills and competencies 
that support their educational needs. 

Programs support learners’ educational needs by promoting progress toward attainment of linguistic, 
mathematics, communication and problem-solving competencies. Progress is demonstrated by 
improvement in participants’ ability to understand, speak, read and write English; to perform basic 
computarions; and to function more effecrively in the home, community and workplace. 



Sample Measures 

• Standardized test score gains. 

• Competency-based test score gains. 

• Teacher reports of gains/improvements in communication competencies. 

• Altemarive assessment methods (for example, portfolio assessment, student reports of 
attainment or improvement in specific employability or life skills). 

Indicator 2: Learners advance in the instructional program or complete program educational 
requirements that allow them to continue their education or training. 

Programs promote progression to higher levels of learning within the adult educarion program or 
promote the attainment of skills required for learners to advance to other education or training 
opportunities. Progress is demonstrated by participants’ attainment of a credential or movement into 
other programs or skill levels. 



Sample Measures 

• Rate of student advancement to a higher level of skill or competency in the adult education 
program. 

• Attainment of a competency cerrificate. 

• Attainment of a GED or high school diploma. 

• Percentage of students referred to other educarion or training progrtims. 

• Percentage of students entering other education or training programs. 
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Program Planning 

Indicators: Program has a planning process that is ongoing and participatory; is guided by 
evaluation and based on a written plan that considers community demographics, needs, resources, 
and economic and technological trends; and is implemented to its fullest extent. 

Planning begins with a written plan that proceeds from the program’s mission statement. The planning 
process is ongoing, with mechanisms for revising plans on a regular basis, drawing on input from 
program evaluations. Planning is responsive to the needs of learners and the community through input 
from staff, students and other appropriate programs and organizations in the community. 



Sample Measures 

• Existence of a planning document that specifies program goals and objectives and is regularly 
i .. v'iewed and revised. 

• Openness of the program to community input through mechanisms such as an advisory board, 
staff meetings, student questionnaires and public hearings (checklist); and frequency with 
which these sources are consulted. 

• Evidence of use of documents in the planning process having data on community needs (such 
as census data or needs assessments). 

• Program plan matches community needs regarding location of classes, skills taught and type 
of program services offered (for example, sufficient ABE or basic literacy instruction). 

• Existence of pimgram evaluation component and evidence that evaluation feeds into the 
planning process. 

• Congruence between planned progr m activities and actual activities. 

Curriculum and Instruction 

Indicator 4: Program has curriculum and instruction geared to individual student learning styles 
and levels of student needs. 

Curriculum and instruction are individualized to meet the educational needs of students with diverse 
educarional and cultural backgrounds. Since students have different learning styles and goals, 
instruction includes a variety of instructional approaches and strategies. To ensure the program s 
success in meeting student needs and capturing changes in those needs, student and staff input is 
obtained periodically. 

Sample Measures 

• Use of student assessment information to inform the instructional process. 

• Existence of studeni goal-setting process linked to decisions on instructional materials, 
approaches and strategies. 

• Instructional content addresses educational needs of individual students. 

• Instructional strategies used and frequency with which they are used, measured through 
observation or self-report. 
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Staff Development 

Indicator 5: Program luv; an ongoing staff development process that considers the specific needs 
of its staff, offers training in the skills necessary to provide quality instruction and incliuies 
opportunities for practice and systematic follow-up. 

The program’s staff development is designed to enable staff to provide quality instruction by meeting 
their specific training needs. Staff development begins with an orientation to the goals and philosophy 
of the program for new staff and continues with periodic training on effective practice. The process also 
includes input from staff and students to identify needs, and pracrice and follow-up to ensure effective 
instruction. 

Sample Measures 

• Presence or absence of preservice and in-service staff development opportunities that include 
a program overview, philosophy and goals of the program and ongoing topics appropriate to 
adult learning. 

• Existence of process for identifying staff development needs. 

• Staff development based on known promising pracrices. 

• Effective staff performance as measured by student ratings or observations of staff. 

• Percentage of staff needs met through training activities. 

• Average hours of preservice and in-service staff development training received by staff. 



Support Services 

Indicator 6: Program identifies students’ needs for support-services and makes services available 
to students directly or through referral to other educational and service agencies with which the 
program coordinates. 

Programs identify support-services needs that affect participation in the program and promote student 
access to these services by referral to other agencies or direct provision of service. The program has 
formal or informal coordination linkages with other service providers to facilitate referral. 

Sample Measures 

• Presence of a process for idenrifying student support-services needs. 

• Presence of agreements or linkages between the program and child-care and transportation 
providers. 

• Number and type of support services provided. 

• Number and type of support services to which students are referred. 

• Percentage of students obtaining specific, needed services through the program or through 
referral. 



Recruitment 

Indicator 7: Program successfully recruits cotnmuniiy population identified in the Adult Education 
Act as needing literacy services. 

Sample Measures 

• Types of recruitment activities the program performs. 

• Percentage of ttrrget population enrolled compared with state demographics. 

• Percentage of students enrolled with specific characteristics compared with the population 
with these characteristics in need of instruction in the service area. 

• Percentage of t;u-get populations enrolled compared with state average. 

Retention 

Indicator 8: Students remain in the program long enough to meet their educational needs. 

Retention is measured in light of student progress towtird meeting their educational needs by time in 
program. Retention benchmarks are established that account for the type of program and learning gains 
expected for a given number of hours in the program. 

Sample Measures 

• Hours in program by type of program and learning gains achieved as measured by student 
progress. 

• Percentage of students returning to the program within specified time period. 

SUGGESTED NEXT STEPS FOR YOUR UNITED WAY 

Order copies of the Model Indicators of Program Quality and United Way’s Assessment of 

Literacy Needs and Resources. 

Using one or both of the above tools, assess your local literacy programs. 

Distribute this fact sheet to your board and identify together your literacy funding criteria. 

S upport or assist your funded literacy programs to conduct a self-assessment of program quality. 



YOUR NEXT QUESTIONS OR NEEDS 

Call Education & Literacy Initiative, United Way of America, (703) 836-7100. 




Prepared by A Iden h^knesister, Literacy ConsuUanty UnitedWay of America Education and LUeracy Initiative, With consultation 
from: Robbin Sorensen, United Way of America; Charles Colvin, Montgomery Area United Way, Alabama; Kathy M^us,United Way 
ofNortheastCarleton County, Minnesota; Sharon Morioka, Uniu^d Way ofSanDiego County, California; Susanne Sandusky, United 
Way of Aroostook, Maine; Jeannette Morrison-Marks, United Way of Oakland County, Michigan; Gcralyn Sheehan, United Way of 
the St. Paul Area, Inc., Minnesota; and Dr. Barbara Alexius, Literacy Consultant, United Way of America Education and Literacy 
Initiative. 
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